"Wikipedia Maps" By Yuri Astrakhan. [Live captions by Norma Miller @whitecoatcapxg] Good morning, thank you. Let's welcome Yuri to OSM and Wikipedia, thank you. >> I'm actually by far worse than ... who spoke with you yesterday. I'll do more of an -- all right, so about a year and a half ago, I came to my boss and said we have this amazing thing called Wikipedia. But for some reason we have no good map support, and even though quite a few of the articles contain some sort of information about Wikipedia, I mean about maps, about geography, about coordinates, we don't have any good service to really visualize it, let people interact with it, and it kind of turned from static pages that Wikipedia frequently to something interactive where you can click and zoom in and explore. So my boss at the time really helped with getting the project started and Max and Julian really helped get it on the road and actually Max and Julian will join me later in the presentation. Anyway, Wikipedia has tons of information. We need to get it somehow visualized and shown to the users to tell a story. Just having a basic map is not actually enough to have an article, because they can just go to Google Maps or any other site that shows that map. They need to see something related to the reading. For instance, if you're reading an article about Don Quixote, you probably want to see how he went through Spain. Or where whales kind of live migrate, and many other things. So now we need to combine the two, the contributory power of all the volunteers to add this wonderful data to the map that OSM created or some community created to be the backdrop to give the context of where in the world this is. And unfortunately, my super slides are not showing the way they were supposed to be showing. There's a presenter thingamajig that I should probably move to -- anyone knows how to do Google presentation? It's like it just keeps disappearing. Oh, there we go. We've got it. we're going to have a full screen. So first we created this very powerful system that Julian is about to start talking about, where using leaflet we gave the ability for Wikipedia to add tags. >> Yes, so I'm going to -- like you guys already know what map is, or already know what leaflet is so I'm not going to go too much into details, but basically we created that new tag in MediaWiki, so people can simply put map frame and the width and the height and the zoom and the map center, and some churches and it will show a map and in the article. >> Another tag that we created is map link, and so that would just put a link within the article when you click on it, it opens up up a map in the first screen like this, and this is something that we -- so basically I'm on the front-end side of it and we have a Javascript API that volunteers can just require and then start playing with it, and so this is an example of customization with WikiVoyage, which is one of our side projects, and you can see that they added some controls to the specific project. But maps are in the articles, but they are very often represented kinds of data and a lot of people are criticizing the search in Wikipedia, and I'm part of the discovery department, which is doing a lot of the work to improve the search on wiki, and they recently I did geo search to the search features, and so this is an example of how we can solve for it queried the articles nearby down near San Francisco, for example, and this is it's going to be in production very, very soon, I feel. And maps are very, very greater, and we can build all kinds of integration, so we have a lot of coordinates and a lot of projects. This is an example on Commons, and this is a picture of the other second picture of the -- between 16, I think? 15? I don't know. So we have coordinates, but you know, again it's really hard, so you can just put a map and it's much easier and much faster to actually see where the photo was taken. In the example in Wikidata, we have a bunch of entities, and they have some coordinate locations associated to them, and so again, this is a very nice example of showing those coordinates within the map and showing that point on the map, and also if you look at the dev history, you can have two maps and you can see where the map moved and you can quickly review if that makes sense on that. And that's another example that Yuri is probably going to come back. >> Yuri: So but what I wanted to share the most with you, to talk about the most, is how two of our communities can benefit from our work together. Catherine already mentioned Wikidata, and I think Wikidata is like the cornerstone of this collaboration. This is the service that was built about three or four years ago by Wikimedia Germany, and it runs on the Wikimedia servers, it's -- it connects all the different articles, across languages, and moreover, it gives a lot of metadata. It stores a lot of data about an article or about an entity that is very easily queriable and indexable. Using Wikidata, we can really bring together OSM elements and Wikipedia information for all sorts of usages and I'll demo it later. This is the very simple query service, not -- sorry, the service is not very simple. I still don't understand it, tell you the truth, even though I've been using it quite extensively. It's kind of like SQL, but totally not. [laughter] If anyone has ever used RDF, you would know what it is. We have one engineer who actually understands it. He's the guy who implemented it. So whenever I need a query, I just ask him. Works out quite well. [laughter] The point is that you can ask very, very elaborate questions, you can say give me a list of most -- the biggest cities with female mayor, or many current mayor or previous mayor or whatever, you can do all sorts of very, very strange requests in terms of any kind of metadata for any entity out there, and if it has GPS coordinates, well, you can visualize it and this is an example of location of castles that are also archeological sites. It's just some random example that I picked. If you go to query.wikidata.org, and click on examples, there's a gazillion of them. And some of them get visualized with maps, some visualized as a table. Actually Katie here is a good representative of a Wikidata table. Going on. This is something that actually we did not do. This is totally not our thing. This is something MapBox people did for OSM editor and published it. There's a link at the bottom if you can see it. This is the first attempt to closely knit together a Wikidata, Wikipedia, and OSM and so when you add Wikipedia, it actually auto inserts Wikidata. I want to highlight, I feel this is a work in progress, even more should be done there, because we should not link to English Wikipedia articles. This is a very in-flux thing. Wikidata is the only thing -- no, it's not the only thing, but it's more likely to stay intact and won't change. Wikipedia articles get renamed all the time. I mean obviously something big like the United States would not get renamed -- or it might. [laughter] I mean we do have a new pres OK, I'll shut up now. But but smaller cities get named all the times. But Wikidata is what should be tying Wikidata and Wikipedia all together. In any language you want to read it in and possibly get all sorts of metadata from Wikidata? Again, do we want to share all the same things in two places? And how to bring them together so that they work together much nicer? And benefit from one another. Some of the benefits of this integration: This is just my other project, it's called graph extension. It allows data visualizations, and here, even though it is usually used for like charts and regular graphs, it also supports mapping, like map information, so you can -- I'll actually demo it in a bit, but the very top one is actually a Wikidata query, requesting all the subregions, basically admin levels is one, from a country. Any country. You give it a country ID, it gets the list of subregions for, then it uses the OSM service to add the geo shapes for each of these subregions. Then it gets all sorts of other data from Wikidata, like population and the language, whatever name you want. And possibly the image and the flags of that subregion, and then it all gets shown in a little graph on the Wikipedia page and all of this technology is live and available everybody with Wikipedia if people want to start inserting it and it's interactive as you can kind of hover over different regions, it will pop up and explain what that little region is. And anyone can edit it and change it. Lastly, I want to talk a little bit about Kartotherian, and Max will help me with this. Kartotherian is a little tile service that we had to build for maps. Because it's a vector tile server. I highly encourage you to take a look. It still needs some work in terms of making it independent of WMF-specific configurations and technologies. But it's a generic service to serve vector tiles or raster tiles. It can be used to store any kind of information and vector tiles such as geo shapes which the previous slide was using. >> Max: Yeah, so in principle, this is a not something architecturally. This is a particular vector tile server, so we have a Postgres database replica of OSM. We use it to pregenerate all the tiles in vector format, at levels 0 to 15. We store the resulting vector tiles in Cassandra. Alternatively we have a Postgres backend and we might experiment further with it, but we were already using Cassandra, and it seemed like a good choice with its multi-data center support. And then there is Kartotherian that, it can just serve you vector tiles from Cassandra, or if you need the PNGs or jpg, it converts them on the fly and in front of our Kartotherian, there is a HTTP caching layer for which we are using varnish. Also, when OSM updates are happening, we have tile regeneration service, that distributes the load across the cluster. It can be used to generate tiles, regenerate tiles, Yuri wrote a pretty sick architecture where it actually resembles how you work with various Unix tools, so you basically pipe one thing into another thing, and into another thing, the same way in Kartotherian there is a system of sources, so actually internally, all the operations are basically copied from this source into that source. Yeah. Yuri: And now before we do the Q & A, I want to do a very quick demo, if I'm able to move that screen to the other screen. All right, can you hear me? Yeah, excellent. So as Julian was saying, this is basically a wiki voyage site and it has different links. You can click on it and the internet is down. [laughter] But you understand the concept, right? [laughter] This is the graph extension I was mentioning, and as you can -- oh, great, see, yeah? There goes the demo. Imagine beautiful pictures popping up as you move this -- oh, there was a picture. Oh, I see some flags popped up. Ah, it's working! Wonderful, the internet is back. So the point is with just this one line with different projections of different configurations, sizes, whatever, you specify which country you want and it just shows it for you. There's another one for you. And -- there's it shows population, for example, right there. All this Wikidata data is available. All the images are available in commons, so all of this can be put together right there in Wikipedia. This is actually Wikipedia site, one of our projects, so this template can be copied as is to another Wikipedia and that we switch to Q & A. [speaking off mic] >> The castle? No, well, the query service itself is simply it just gets whatever Wikidata data is there, but if you pump that query through the Kartotherian service, Kartotherian service can append geo shapes for the Wikidata IDs, so if the element is tagged in OSM database with a Wikidata ID, Kartotherian service can just convert that query from Wikidata to a JSON with the needed elements, plus all the properties of the original query. Yeah? AUDIENCE MEMBER: Actually I'm going to need -- [speaking off mic] >> Yuri: Yeah, I'm sorry. Google preparations need to be adjusted a little bit, I guess. Sorry. Ask me questions that would be easier, probably at this point. Questions? AUDIENCE MEMBER: Do you have any special handling for ways for transit paths and things -- points and polygons seems kind of biased towards that, do you have any -- >> Yuri: Max, you want to take this one? Max was doing the data conversions, so -- >> Max: Have you tried the actually pulling lines out of it? >> Yuri: Are you talking about geo shape service? Sorry. AUDIENCE MEMBER: A whole data question regarding things like transit lines or -- >> Yuri: Oh, I'm sorry, my bad. I thought you meant the data. At this point it's querying just a table with polygons, if the other table is available, I mean it will be fairly easy to change that to extend it to lines or whatever other elements there are. I see, like it should be very straightforward and simple. AUDIENCE MEMBER: You can represent like relationships between stations and then each of them as a coordinate to like you can make a map that way, like from the network? >> Yuri: Mm-hm. AUDIENCE MEMBER: That's a different way to do it. Yuri: I would love to somehow figure out how to get that information and plot it more easily. Yes? AUDIENCE MEMBER: Are you thinking about not problems which are noticed within Wikipedia being fixed in OpenStreetMap. Yuri: This is something I actually forgot to mention, but like how do we drive all the volunteer help orders OSM, is that the question? So given all these amazing volunteer support that Wikipedia gets, we would really love those volunteers to also be able to contribute to OSM and I think we should get closer ties, either like joint log-in somehow, ability to click a button and just kind of notify that there's a problem with a certain map, or go directly into OSM iD editor or other editor and contribute. Would love to work on this, this is one of our main goals to improve the maps this way. >> Julian: I just want to add this is one of the main goals and we have a ticket fabricator to basically get some ideas on how we could implement that I feature and how that expands with work. So if you have any ideas on how we can put a repo together or like contribute back to OpenStreetMap, how that would work, if you have any ideas, come talk to me and let's try to figure out something. Thank you. AUDIENCE MEMBER: So do you have plans to add like a time access to the Wikipedia map, like for example, give me a -- give me how the world looks like in the 10th century, so because on Wikipedia, there's lots of like historical data, and it's historical view of the world is very important to us, so I'm wondering if you have plans for adding a time access to the map. Thanks. >> Yuri: Trying to find a demo for you. Right, so historical -- not prepared to show you a demo, but the historical map questions have been coming up quite a lot. There is rudimentary support in graphs to show geo changes over time. We do not have it for support in general map. >> We do have a but like there's like the map system that was like originally developed like for the New York public library, but we have like in the instance of that, like that's integrated or being integrated with Wikimedia Commons, so you can upload like an old map and then like warp it, and I think there's some integration with iD editor, like it can be digitized and I think they're working with like open historical map or something, so to be able to have like historical, like, data, it's like using the OSM technology, so like, I'm sure at some point, like we'll have it integrated with this, but like there's a a bit of work being done on that, so. >> Yuri: So, yeah, and oh, -- yeah. Oh, we're out -- sorry, we're out of time. Come talk to us. We're here all weekend and thank you for coming. >> Thank you, Yuri, thank you, Julian. [applause]